Initialization Schemes and Gradient Clipping
These images were originally published in the book “Deep Learning with PyTorch Step-by-Step: A Beginner’s Guide”.
They are also available at the book’s official repository: https://github.com/dvgodoy/PyTorchStepByStep.
Index
** CLICK ON THE IMAGES FOR FULL SIZE **
Papers
- Xavier/Glorot Initialization: Understanding the difficulty of training deep feedforward neural networks by Glorot, X., Bengion, Y. (2010)
- Kaiming/He Initialization: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification by He, K. et al. (2015)
Vanishing Gradients
Source: Chapter Extra
Initialization Schemes
Source: Chapter Extra
Comparing against BatchNorm
Source: Chapter Extra
Gradient Clipping
Value Clipping
Source: Chapter Extra
Norm Clipping
Source: Chapter Extra
Using Hooks
Source: Chapter Extra
This work is licensed under a Creative Commons Attribution 4.0 International License.